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ABSTRACT 

This report reviews evaluations of two instructional 
systems that the faculty of the College of Agriculture, University of 
Illinois, are now using in their extramural teaching. The systems are 
('’) "Univex Net,” which transmits audio and visual signals via 
telephone lines from one campus classroom to another classroom 
located somewhere else in the state, and (2) auto-tutorial carrel 
units which provide for independent study. A combination of the 
auto-tutorial and the Univex system is also evaluated. The 
evaluations are learner oriented. Attitude was measured with the 
Illinois Course Evaluation Questionnaire. Actual performance was 
measured by grades and specific criteria for success in obtaining 
educational objectives. (The questionnaire and four references are 
included.) (author/ly) 
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During the past decade the concept of an instructional system has teen 



offered as the great promise for the solution of complex adult education 
training problem^. The use of the systems approach for extramural instruction 
poses a valid research problem regarding the effectiveness of such systems and 
their impact on the learner, the teacher and the educational institution that 
uses them. The opportunity to seek answers to such questions presented itself 
when the College of Agriculture, University of Illinois started teaching extra- 
mural courses by the UNIVEX JJET in the fall of I 968 . 

The UNIVEX NET is an educational system developed by the Division of 
University Extension that provides extramural courses by telephone to educa- 
tional centers throughout Illinois - Each center instantaneously receives both 
written and voice communication from the instructor. In addition, each student 
in the off-campus class has bhe ability to write and speak to the instructor or 
his classmates at different locations. Thus, all elements of a normal class- 
room are present except the physical presence of the instructor. 

One point needs to be clarified. Most systems acquire the name of the 
media or hardware of the system. Media are not instructional systems but 
the tools by which the instructor and his institution can facilitate the 
greatest of all acts of communication , . . that of an instructor with his student . 
and between the student and his instructor. It is this act of communication 
that we should attempt to research not the hardware. However, we must recognize 
that the hardware facilitates the development of complex instructional systems 
which heightens the impact of the instructor in controlling the instructional 
system of which he is a part. 
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If "instructiQMl" can be defined as those elements which promote learning 
and '’system" defined as the many components unified as an entity to make possible 
a process, then, for research purposes an instructional system can be defined 
as that entity which makes possible the process of education for any given subject 
matter . 

A basic assumption can be made that there is nothing in an instructional 
system, of and by itself, which produces improved learning by the participant. 

If poor instruction is fed into the system, it is still poor instruction when 
it reaches the student. If the learner is not capabl'd'of learning, the system 
can do nothing about his capabilities . 

It is difficult to cut-up an ongoing instructional system into meaningful 
variables. If this is attempted in a typical experimental design, the system 
itself is destroyed and the sum of the parts are much different than the whole. 
Another limiting factor is that students and instructors will not be manipulated 
like varieties of corn, pigs or cattle in order to create the proper experimental 
design to measure the contribution of such variables as the hardware, software 
and the learning environment. 

Thus, the researchable questions must deal with a system at work. Research 
tools must be limited to those useful in conducting field research and case 
study research. Conclusions from such investigations must be limited to a 
specific system, perhaps to a particular kind of classroom and to a particular 
kind of student, for a particular kind of hardware and software and for a specific 
instructor. Thus, the results are not expected to be generalizable to other 
instructional systems. However, they may be useful as guidelines in evaluating 
the performance of other systems. 

With the current popularity of teacher evaluation end the availability of 
proven test instruments, it soon became appai’ent that this would be a fruitful 
approach. Others support this rationale. 
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Tyler (2) indicates: 

Students can report on their interest in the course, on their under- 
standing of what is expected of them, on their satisfaction with 
achievement in the field, on the amount and extent of their study, 
and the like. There are, of course, other important aspects of 
teaching which the students are not in good position to judge, 
such as the soundness of the objectives, the validity of the refer- 
ence material provided, the relevance of the approach. On the whole, 
hovrever, it has been found that the summation of student judgements 
obtained from a questionnaire is positively correlated with other 
evidences of effectiveness of teaching, , . . 

It is in Wientge’s (4) documentation of "Adult Teacher Self-Improvement 
Through Evaluation of Students" that the research value of rating scales can 
be seen. His use of such rating scales in an instructor rating form lead to 
the conclusion that the Illinois Course Evaluation Questionnaire developed by 
Richard Spencer and Lawrence Aleamoni (l) would be a logical research instru- 
ment. Wecke (3) reports the successful use of the CEQ with extramural courses, 
also « 

Spencer and Aleamoni assume that if one does in fact change student 
behavior, in the specified direction, as a result of an instructional system, 
then that system has been effective. If that system has been effective then 
there could be a large number of elements in that system contributing to its 
effectiveness, such as the instructor, textbook, homework, course content, 
method of instruction, student interest, student attention, general student 
attitude towards the course, etc. Assuming that all of the elements directly 
or indirectly affect student behavior in a course, and assuming that students 
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are the on3.y ones who are constantly e^cposed to those elements, they appear 
to be the most logical evaluators of the quality and effectiveness of the 
instructional system. 

The 50 item CEQ is scored using a weighted point system, under the 
assumption that students who indicate strong responses to questionnaire items 
should be differentiated from those whose responses tend to be more moderate. 
All responses to the CEQ items are based on a common scale, from Strongly agree 
(SA), through agree (A), to disagree (D), and strongly disagree (SD). There is 
no neutral position. A positive and a negative item expressing roughly the 
same concept foarni a pair. The scoring of these items in match pairs is useful 
in identifying the "careless" student responses. An example of these pairs is 
Item "More courses should be taught this way" and Item #37 would prefer 
a different method of instruction." 

Factor euialysis (Thixr stone, 194?) was used by Spencer and Aleamoni to 
determine the six sub-scores: 



Factors Obtained from the Final 50 Item Questionnaire (l) 







Factor (gub-Score) 


Number of Items 
In the Factor 


$ Variance 


I 


General Course Attitude 


8 


30 


II 


Method of Instruction 


8 


6 


III 


Course Content 


8 


5 


IV 


Interest and Attention 


8 


4 


V 


Instructors 


8 


3 


VI 


Other 


10 


3 
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In checking the reliability of the sub« scores, Spencer and Aleamoni 
concluded that general course attitude, method of instruction, interest and 
attention and instructors were considered acceptable. The items defining 
content and other sub-scores were the least reliable. They indicate that a 
few of the content items correlated with general items, which explains the 
lower internal consistency of these items. 

In the scoring procedure, the average item response is computed for each 
item for a given instructional system (class). The instructor’s mean by item 
is then compared to the total results across all sections tested in the standard- 
ization population and decile norms are printed for each item mean. A total score 
and a set of sub-scores are also computed and presented in the computer print out 
report. The total score is the mean response over all questionnaire items. The 
sub-scores represent definite areas in an instructional program that can be 
considered relatively independent from each other. For example, the content 
of a course may be rated by the students as good while the method of instruction 
may be considered poor. 

To date, approximately 250 courses with a total in excess of 100,000 
students represent the University of Illinois normative population. Normative 
data for each item, expressed in deciles, is based upon the responses of the 
total normative population whereas the normative data for the sub-scores is 
also reported by department, course level, rank of instructor, etc. 

The Illinois Course Evaluation Questionnaire is simple to use. Copies are 
printed on a Digitek Answer sheet and supplied by the Measurment-' and Research 
Division of the Office of Instructional Resources at the University of Illinois. 
Each student responds by marking directly on his ovn answer sheet with a 
conventional graphic pencil. It normally takes about 10 minutes for completion. 
Administration of the questionnaire is supervised by the course moderator or 
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test proctor at each educational center on the UJIIVEX NET and the completed 
forms are inserted into an envelope for mailing to the researcher prior to 
administration of the final exam. 

Prom the returned forms the following is produced: 

A. A print-out which indicates average sub-test and total scores, 
and the norm decile, 

t 

B. A print-out which includes specific item responses, their means 
and the norm decile. 

Pioneers in our efforts with the instructional system involving the UNIVEX 
WET were two instructors in Agronomy during the 1968 fall semester. This gave 
us a unique opportunity to observe graduate -level student attitudes about this 
course in four different locations, two of which would be on the UWIVEX WET and 
one conventional classroom on campus and one extramural classroom located 150 
miles from the campus to which the instructor travelled each week to meet the 
class. Instructor A taught this extramural course in person. According to 
student attitudes as measured by the Course Evaluation Questionnaire, this 
was the best rated course, with the UWIVEX WET course at Malta in second 
position and the UWTVEX BJET course at Freeport in third with the on-campus 
course in last position. Wote that for Instructor B, his highest rating came 
from a UWIVEX WET location. In this assessment from the CEQ results w€' note 
a difference in instructors and location, A surprising evaluation of instructor 
attitude indicates that Instructor B gained a great deal more satisfaction from 
’’face-to-face” teaching than from the impersonal teaching through UWIVEX, yet 
his best rating came from a UWIVEX location, which was even better than Instructor 

A 

9r. The rating at Freeport was significantly lower than the other three locations. 
In these ratings we obiierve the complexities of an instructional system as 
they are uncovered by the CEQ. 
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Another course, Pork Production 303, was taught during the 1969 spring 
semester providing the first opportunity to see how auto-tutorial and UNIVEX 
systems would work together for a graduate -level class. 

In observing the sub-scores obtained from the Course Evaluation Questionnaire 
we find the course ranking in the 8th decile in general course attitude which was 
also the decile score for the instructor. However, the decile for method of 
instruction was at 3, which was two points below the all -University average. 

With all measurements considered, this particular course with this particular 
instructor and system rated better than the all-University average. 

The students were taking the course at sir'educa'tipnal centers throuighout 
the state. At each center there was an auto-tutorial carrel tied into the instruc- 
tional system with 10 auto-tutorial units assigned by the instructor. During 
the course we decided to document the use of these carrels as in a case study 
approach to see if the students were using the units and if so whether they were 
encountering any problems, and at the same time to get their evaluation. We 
noticed a great deal of difference in participation at the various centers. 

Eight of the units had been previously assigned by the instructor. One-sixth 
of the students had not on their own studied the units by that time. At Decatur 
the AT carrel was located in the Cooperative Extension adviser’s office two 
miles away from the educational center. This definitely hindered the study of 
the AT units by the student. The students were not motivated to travel this 
distance. Again, the times that they could get access to the carrel were not 
appropriate times for them to study. Note that in the other locations half or 
more of the units had been studied where the ^f^-tutorial carrel was located 
at the educational center. This immediately leads us to conclude that the carrel 
must be located at the educational center for the convenience of the student. 
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Except for Decatur alX locations were favorably inclined towsurd the need 
for auto-tutorial units in the course. For two locations, access to the auto- 
tutorial units was a limiting factor in studying them. Carrel equipment 
problems did not appear to hinder study except in one location. In regard to 
the question "I would not hesitate to enroll in another couirse using the same 
teaching methods.” we found three locations favorably inclined with three 
locations not favorably inclined. At this time we could perceive the lower 
rating for the instructional method on the decile score mentioned above. 

Also, during the spring semester Agricultural Law was taught. The 
instructor used the conventional system for 57 students on campus and the UIHVEX 
system for 31 students. All of the UNIVEX locations were averaged to represent 
the UNIVEX NET. These included the previously mentioned locations except Decatur. 
The Decatur location was so noisy and distrubing that the students travelled the 
extra distance to Urbana after the first week, to attend the course. In this 
case, the students were with the instructor in the UNIVEX terminal in Urbana. 

The instructor found this was very helpful to have the students right before 
him in conducting the class on the UNIVEX NET. Tests to check for significant 
differences were run between the UNIVEX NET group and the Urbana conventional 
classroom. The results indicated that the Urbana conventional classroom was 
rated significantly better than the UNIVEX NET as to (a) method of instruction, 

(b) student interest and attention and (c) the instructor c The UNIVEX NET was 
not preceived significantly different from the conventional classroom regarding 
(a) general course attitude (b) and course content. However, it should be noted 
that many instructors would be happy with the evaluation received by the UNIVEX 
system. Note that the rating for the method of instruction and course content 
was still above the average for all-University courses. 
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When analysing the course by location we note that the 10 students at 
Malta rated the course quite differently than the 21 at the other locations » 

Wfe assume this Malta group interacted with the system quite differently than 
the other groups and therefore believe group dynamics play a big part in 
attitudes toward the net. 

Achievement of the 31 students on the NET and the 57 conventional class- 
room students were almost identical. Three UNIVEX NET students made higher 
grades than any one on campus. The instructor observed that agricultural law 
can be effectively taught by the system. Much can be done to create "rapport" 
with the students and the lack of visual and personal contact tend to become 
less important as the course prog ses. Adults such as those in this course 
are appreciative of the instructor's efforts, study their assignment surpris- 
ingly well, are attentive and interested and alert, and render .a good account 
of thcmnalves on quizzes and exams. 

The employment of comple>:. instructional systems to implement extramural 
teaching must be accompanied by valid research efforts which can evaluate the 
effectiveness of such systems. Until more effective research tools can be 
developed, the course evaluation questionnaires can provide information about 
general course attitude, method of instruction, course content, interest and 
attention, the attitude toward the instructor and other dimensions of the 
Instructional system. When supported by case study methods which indicate the 
weaknesses and sti-ingths of the system in specific detail, the research yields 
helpful insight about the instructional system. 
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learn more v/lieri other teadiinj> methods are used. 



It was a waste of time. 



Overall, the course was good 



The textbook was very good. 



The instructor seemed to be interested in students as persons. 



More courses should be taught this way. 



The course held my interest. 



I would have preferred another method of teaching in this cour; 



It was easy to remain attentive. 



The instructor did not synthesize, integrate or summarize effectively. 



Not t.iuch was gamed by taking this course. 



The instructor encouraged the development of new viewpoints and appreciations. 



The course material seemed worthwhile. 



SAMPLE MARKS; 



USE 

PENCIL 

ONLY 



a) 

b) 

I 

d) 



RESPONSE CODE 
MARK 
MARK 
MARK 
MARK 



1 F YOU .STRONGLY AGREE 
Y/lTH THE ITEM 



IF YOU AGREE MODERATELY 



VVITH THE ITEM 

LiISAGREF MODERATELY 
WITH THE ITEM 
IF YOU STRONGLY DISAGREE 



WITH THE ITEM 



It was difficult to remain attentive. 



Instructor did not review promptly and in such a way that students could understand their weaknesses. 



Homework assignments were helpful in understanding the course. 



There was not enough student participation for this type of course. 



The instructor had a thorough knowledge of his subject matter. 



The content of the course was good. 



1 F PART II OR 
MARK HERE — 



IS TO BE USED 



The course increased my general knowledge. 



The types of test questions used were good. 



Held my attention throughout the course. 



The demands of the students were not considered by the instructor. 



Uninteresting course. 



It was a very worthwhile course. 



Some things were not explained very well. 



COMPLETE SECTIONS BELOW ACCORDING 
TO YOUR INSTRUCTOR’S DIRECTIONS: 



OPTIONAL 
PART II 
ITEMS 51-75 



OPTIONAL 
P/RT III 
ITEMS 76-100 



The way in which this course was taught results in better student learning. 



The course material was too difficult. 



One of my poorest courses. 



Material in the course was easy to follow. 



The instructor seemed to consider teaching as a chore or routine activity. 



Course material v/as poorly organized. 


^ O 




Course was not very helpful. 


W s 

P O 




It was quite interesting. 


jn 5| 




1 think that the course was taught quite well. 


j--r miA 




1 would prefer a different method of instruction. 


Cb ^2 ^ 

T3 




The pace of the course was too slow. 


lO 




At times 1 was confused. 


^ CM -H 

O m "P 




Excellent course content. 


UJ T3 




The examinations were too difficult. 


Lu ^ 




Generally, the course was well organized. 


Pi p 




Ideas and concepts were developed too rapidly. 
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i 



;■ 



■4 

o 



Some days 1 was not very interested in this course. 



It was quite boring. 



The instructor exhibited professional dignity u bearing in the classroom. 



Another method of instruction should have been employed. 



y 



The course w,ns quite useful. 



would take another course that was taught this way. 



